Skip to content

Improve progress reporting#32

Merged
HappyTetrahedron merged 1 commit intomasterfrom
local-backfill-changes
Oct 29, 2024
Merged

Improve progress reporting#32
HappyTetrahedron merged 1 commit intomasterfrom
local-backfill-changes

Conversation

@HappyTetrahedron
Copy link
Copy Markdown
Contributor

Summary

  • I changed the log line that reports on progress after each query run to include more useful details. These are particularly needed for the use-case of backfilling data over a larger timeframe and across multiple products - it needs to be visible at all time where the processing currently is, so that it can be restarted in the case of transient errors.
  • I furthermore removed the printing of the query results. It is extremely spammy, printing up to a hundred results for each timestep, making the progress report log lines impossible to find (or keep in scrollback buffer) while providing little actual utility.

Checklist

  • Categorize the PR by setting a good title and adding one of the labels:
    bug, enhancement, documentation, change, breaking, dependency
    as they show up in the changelog

@HappyTetrahedron HappyTetrahedron added the enhancement New feature or request label Oct 29, 2024
@HappyTetrahedron HappyTetrahedron requested a review from a team as a code owner October 29, 2024 12:20
@HappyTetrahedron HappyTetrahedron merged commit 2079fd1 into master Oct 29, 2024
@HappyTetrahedron HappyTetrahedron deleted the local-backfill-changes branch October 29, 2024 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants